DCCnHEIil EiSOBE 

Chanaiet, John Si*;; DeLx.ti^^^ 

A: ;M€thcd;clo.gy_ ictf i^^^ Iiif crinaticn System 

be'sign; - _ . : . ^ 

Ohxp^-State- -Cpiiv^. pciuibusi CCBEUter. -and^ InfortatioB 
"Science :E;fs.€k 

Nati^)nai :Sci€iiC6, rEaundaticr ^ Hasliiiig tbii, E. C. 

6s;u-tiSiiC-Tir7^^^^^ ' ■ ^ ^ ■ 

JiiD 77 ^ 
•Sis75-21648: 

aF-$P..83 :HC-:$25 06^ ,Eius. f csiag€ . 

Eesigners;. *D4sigt pijet^ EvalwaticDc Criteria; 

♦Evaltfatiph SfiiE^^ fipfl prie^^^^^ 

N,eed£; *iiif ctmat S.ysteis^^ Ipf crmait^^^ ptia^iiiat^ 
^♦EerfprmanceCt^^^ lerf.ciaapcej^^ *Proc^tam^ 
^Design ;: .bse. :S$^di4|- ^ ' " ^ .1 - r ^ 

"*BSIRACr3^^^ « . / 

^ ihe cpiJEiipk:i:ty cf th^e desigl p_i:cblejn for iodern 

cpip.utex based inf proatio has iiacr €as€;^ 

^ts pred€C€;ssor^^ iBje :prqblismj prfes.ent^ 
configure a system >'hic^^ crit|rlcD whil% ac^ 

; sjste^^^^ related iperfcrn^nce diit^^^^ The puip^^^^ 

^piper. xs to ^pi^sfni an^evaluat^^^ the 
^problei^ c^^^ €.Valui,tic|: c| iifcrmaticn 

; systems i The goal of this t^chniqne^ %k . tP E vi^!^ the deligjaet with, 
an; eyai^fe XI) seeks tp estktlish i caisal 

reiat^^ Be^iieen user goal attaiEjieKts aid systett res^u^ 

expehd^^^^^ Mhil4 :ailciii^^^ for the ecil^citidn p^ to 

obtSip; e inf oraatipn; (2) measure^^ iipatt cf iidiy^^^ 
user classes on internal system ^gerfp anc (3): idepti^ those 

information iystej acfiyiti^s wliph shciild be mcdified to improve 

, ^ystem^^p^^^^ tp thbse gtai:sf Ali^^c^^^ achieved by 

maiii^ainin respurce utilizatioj statiltics en a user, class bksism it 

_is>vthpug fram€wor:k is; cajati 

eriminati^^^ of the numerous ncnrsatis^ 

directing ihe designer to £h0 most advantageous cnes. (Authcr/DAG) 



BD 153 618 

AUTHOR 
•TITLE 

^ SEpNS .AGENCY 

ipiifl MtI 

^GRANT^ 

. spfl" 

!EDRS FHiCE 

:bi4c|ip^ 



. ;Re:pfoduc^o that cap, te made * 

* . ' ^ ^ , \f xok the pri^if^l Sctoment. * 



ERIC 











L:.. ^ :osu.T.cis|c^ ^ 






U.S. OiPARTMiNT OF NIAtf N; 
tOUCATIONAWItFARI^ 

NATIONAL INSTITUTt OF * 
^ " ^fOUCATIWI^ ^ . ^ \. ^ _ " 










' THIS. DOCUMENT MAS-8EEN REPRO:^ 

OUCEO. EXACTLY- AS.RECEIVEO^FROM 
^ THE PERSON OR ORGANIZATION ORIGIN* 

ATING IX' POINT S.OF^yiEW OR OPINIONS - ^ . 
. '5 TAT E 0; Ob; N OJaN EC ESS AR IL R E PR E- 

SENT OFFICIAL NATIONAL^INSTITUTE OF 
i EOOCATIONvPOSlTlON OR POLICY. 


'! 


IsH^;.: 




4u!>. .„ , 
















"A :MetHpdology f or MultirC^^ 




\- - ; 


Inf ptiia.tiqn Sys tern Design- ' 




■ r 

! - ■• ; 


by 






: John S Chandler and; Thpmas- G;. DeLiitis 


• K -ft 




"P.^pMlSSION- TO REPRODUrP f uic 
MATERIAL 'HAS:BEEN£GRANTEb,BY 






Celiarina Taylor 


i 
\ 




TojHE ;eoucati6nal.res6ur^^^ 

Kcb^'^-'^'^'^ CENTEI^ JER,C,:aND. 
j USERS OF THE. ERIC SYSTEM.?' " - 






Work performed under 
: Grant-1^ 


f 








^ J-iic iiciujLUiiaX OuXcuCc rOUnQaLXOIl 


i 






j 

A 








|i: jO^- • • - 


i ' 


> 




Computer and^ Inf otiMt 






CotxmJms 






June, 1977 








' - ' • ^"h 


t . " ■ : 








t 

' " " ■ " - ■ ;!2r - 1' 


- i 

' — - i 


1^^ 1 KJ^ ; 




. i 









TABbE OF CONTENTS 



i. ^Iritr.oductfph 



:2\ Evaluative Requlxenierits 



3.i, Formulation of the. Stage 2 Evaluative Procedure' 



4. Liaison id.th;,Stage. 1. 



5; ;Stage 3 Analysis 



'6. Ah Ex2^ ^"""^^ 

^ C 0 he lu s ion: 
^REfERENCES, 



15 



18 



20; 



.-..^24^ 
26 



I ERIC 



PREFACE 

"This work was supported by Grant Number SIS75--21648 from The 
National -Science Foundation to Dr. Thomas G. DeLutis, Associate Pro- 
f essor of Computer and Information Science, and conducted at the 
Computer and Information Science Research Center of Ihe Ohio State 
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of Computer Land Information Science. The , research contract was^ 
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1. Intrqductlori 

, ' ^ J In,,March.^o£ 1973, ;ACMa and NBS' Spions^^^ ' ^oh: computer 

performance evaluation. One of the inajbr results of that Workshop .was 
a. consensus, that there^ been two separate approaches to the evalur 
atloii of information systems performance — one which focuses on the 
computer system domain and the other whose attention is directed at the 
application sys tan (user) domain. Each has its own. gqals and * 

^PPPy^^F system domal^ based oii r 

queiielng and utiilzatlon statistics and. the user domain is evaluated! 
through the perforaance of -requested Measures: such as through- 

put and response- time are connnon for the latter. The Workshop also \ 
concluded that any performance analyses "should recognize both the 1 1 
costs pf a computer instailation and the needs of users for service". ; 

The complexity of the design problem for modern computer based j 
infojraatlbn systems has increased signif icahtly over its predecassbrs * 
due tp:: ^.,-1^ — ^ 

a. the. servicing ofl an expanciihg range of user of uses with 

corresponding diverse .performance goals and resource require-^ - 
mehts, and 

bi the dynamic and unpredictable behavior of the system, as a 
function, of design decisions and' load mix. 

Thusi it' is quite possible to improve ^the performance of the system 
with respect to one or more users at the expense of othersl Likewise, 
because system resources are used by different users. Improving the 
performance characteristics of one or more resources for the benefit, of 
specific, users' may have ah overall detrlmentai effect, on performance. 
The problem presented to the designer is to configure a system which' 
satisfies the user criterion while achieving system resource related 
performance criteria. 
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This was one in a series of Workshops sponsored jointly by ACM and 
NBS to ^examine the major issues involving: computers. Performance 
evaluation, was chosen as the topic of this Workshop because of its 
significant impactron computer usage. A summary of the conclusions 
appear^s in BOE^^ 
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A computer based information processing system can be viewed as a 
symbiotic relationship between the system's users and its hardware, softr 
ware^ and data resources^ Ideally^ the system- will perform "optimaHy^'^ — 
when.it achieves its user oriented objectives within a minimum cost system. 
However, optimal solutions are seldom achieved when systems are complex,, 
ill defined or constrained for reasons outside the control of the designer, 
and thus, the designer usually settles for a satisfactorially behaving system 
Hopefully, systematic procedures are employed to achieve system conf.igu- 
rations which concurrently meet the user objectives while obtaining effi- 
cient utilization of its resources. Current evaluative technologies focus 
on only one criteria in the system design equation, either the user's or 
the system's resource performance. The ability to simultaneously ascertain 
the impact of :resource performance on user goals or vice versa is not 
readily achievable through these methodologies. The purpose of :this 
paper is i6 describe a methodology which establishes a formal liaison* be-r^ 
tween the evaluation of user goals as a function of system behavior and 
the analysis of resource performance as a function of user actiyityV; 

User oriented analyses with objective functions based on response 
time, throughput, and cost have. been| (and are continuing to be) reported 
in the literature. Most frequently, janalytic approaches use queueing 
models as their basis (GAV76, BUZ73, !NEI76 are representative of this type 
of analysis). Due to. the necessity to maintain tractable models many 
siinpiificatlons are required for a model's analytical solution; Simu- 
lation models have also been applied! to user oriented analysis (CON65, 
RpE7P) * Unfortunately, these models yield ohly average and/or aggregate 
_ measures of system response. As a resiiit of these simplifications, the 

analyses produced by both of the approaches fail in many cases to identi- 
fy the relationship, between users and resjources. Therefore, they are 
suspect when used to predict the impact on system performances of modi- 
fying the current environment. 

Alternatively, performance analyses can be made^from the system's 
standpoint, treating the user .and :his goals in the aggregate. The most, 
common approach is a subsystem study, where a particular part of the 
Information system complex is Isolated, witfi the subsystem.. user (s) repre- 
s^^^fc^d. by a stochastic generator, both analytic and simulative. The most 
OTphasized areas of research has been the I/O subsystem (ABA68, NAH73, 



SHE76, HEL76) and CPU utilization (KLE72, AGR76, LEI^71) . The problem 
* with this level of evaluation is that, although providing valuable 
local intuitive insight ^ these models rarely relate to the ultimate 
information system user, and, therefore, do not provide realistic in- 
sight into global performance. 

Examinations of complete systems have also been made^ Exhaustive 
hardware/sof tware measurements have been analyzed by Gonzales and Cantrell 
(GON76, CAN68) while simulation models, including an aggregate user 
.?9HR9nent , been ^built .by^Reexes^ and .Pooch,. Nor land .and. JiUm, XREE75, 

N0R71, LUM70). Although results of the evaluations include resource 
utilization statistics and user oriented measures such as response time, 
there is little attempt in these models to relate particular resource 
usage to the. effect on user goal attaiiuneht. (Two exceptions are 
Lindsay's study of the KRONOS system (LIN76) and Hall's data base. investi-« 
gations (HAL74).) But from practTical experience it is evident that there 

is indeed a relationship between user goals ^and resource usage. In fact, 

I 

?u?§a.jByZ7^)_ha^_rec^ computer 
performance which relate resource activity t'o global system/user measures 
such as response time and throughput. I, 

It is assumed that the objective of good system design is to satisfice 
both performance related criteria. However, in light of the complexity 
of modern systems, many design decisions tend* to be made ^without proper 
supportive evidence on performance. The crux of the problem is to ex-- 
tablish^a causal relationship between user goal attainment and system 
resource expenditures. The methodology to be discussed has been designed 
to establish such a liaison and will be shown to allow for the collection 
of heretofore hard to obtain evaluative information. The methodology 
measures the .impact of individual user classes on internal system per- 
formance and identifies system bottlenecks which inhibit the attainment 
of user goals. This is achieved by maintaining resource utilization 
statistics on a user cla.^s basis. This methodology presents an evaluative 
framework which is capable of eliminating many of the numerous non- 
satisfactory designs by directing the designer to the most advantageous 
ones.^ This methodology is an iterative one with each iteration involving 
three separate but integrated stages. Figure 1 illustrates the activities 
for an iteration. Briefly the responsibilities for each stage shown in 
this^ figure are: 
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Identify 
Performance 
Obj ectlves 



Formulate 
First System 



Stage 1 



Evaluate 
System 



Stage 2 



Evaluate 
User Goals 



Stage 3 




Figure !• Stages In System Design Process 



Stage 1; System Evaluation 
This stage is responsible for evaluating the behavior of a specific 
information system model. It does this by associating the hardware, soft- 
ware and data activities belonging to a specific design with the system's 
user activites. The outputs of Stage 1 are performance statistics ?or 
;the resources in the aggregate and for their behavior with respect to 
identified users (or uses) . To perform^ this function, the IPSS Simulator 
is employed . 

Stage 2; User Goal Evaluation 
Stage 2 has twp^.i>urposes, the first is to ascertain whether the user 
goals are being either over or under achieved. The second purpose is to 
determine the "best" set of guidelines for altering the current system 
configuration in order to obtain the user goals with minimum penalty for 
either undef'^or over achievement. Multiple goal programming is used for 
this purpose. As will be seen, "best" is a function of the assigned 
penalty coefficients in the goal programming objective function. 

Stage 3:/ Design Evaluation 
Stage 3 has two functions. The first is to ascertain whether or not 
the current design's performance is satisfactory with respect to both 
the user criteria and the system criteria. If the design is not satis- 
factory, then this stage's second goal is to define a new system based 
upon the current design, prior alterations, and the results of the Stage 1 
and Stage 2 analyses. Heuristic procedures are currently employed for 
Stage 3. ^ 

The focus of this paper is on the^Stage 2 formulation and its formal 
liaison to Stage 1. The paper also identifies the unique features of 
lESS which permit this multi-stage multi-criteria methodology to be 
achieved . The paper concludes with a discussion of the use of the 
Stage 1 and Stage^.2 results in the Stage 3 heuristics. 



IPSS is a special purpose discrete event simulator whose develop- 
ment was conducted with the support of the National Science Fouhda- 
tioni Grant No. GN-36622. 
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2. Evaluative Requirements 

. "P^LPS?^^ of thisjnethodolog^^^ ari information is. viewed 

as including its users and their goals, and the system^s services and their 
subordinate activities. This is illustrated in Figure 2\ It is assumed 
that the system *s analyst can identify and classify the system's users 
according to their service, request characteristics and according to the 
performance constraints imposed upon the system (i.e*, goals) when 
honoring their requests. It is also- assumed that the analyst can identi- 
fy those information systan activities v/hich are critical to system^ 
performance. Obviously, the complexity of the problem is increased sub- 
stantially when a system supports diverse users or\pro„vides a wide spec- 
trum of services. PJhether or not the system is complex or simple, the 
criteria for identifying system activities should be based upon the 
sensitivity of the system's performance with regard to changes in their 
behavior. 

Information system services are viewed as being^ a .series of distinct, 
yet interconnected activities which are invoked during the processing of 
a stream of user requests for the service. Again, Figure 2 illustrates 
this view of a system. Most likely, system activities are aggregations 
of one or more traditional computer system functions that perform the 
following tasks: ^ 

1. request (job) scheduling, i4 <^ ' 

2. ^task management, 

3. resource allocation, 

4* secondary storage I/O processing, and 
5. application processing. 

The choice of what constitutes an activity is .part of the art of perfprmance 
evaltiation, however, a necessary condition for their selection is ^that they 
be measurable and that these measurements distinguish the service rates for 
separate classes of system services. It is also assumed that the role of 
per:f ormance measurement is to determine the current processing rate for 
the jth activity with respect to the ith service. 

Figure. 3 is a schematic of the functional composition of system 
activities. Each is viewed as an individual queiieing system containing 

10 . . 
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Figure 2. The Methodology's View of an Information System 
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one or more priority queues and one or more identical servers* Addition- 
ally, the performance measure for the activity in processing a request 
-type is the sum of both the queue performance and service functions of the 
activity* Throughout this paper,, the variable RjCi) is employed to identi 
fy this performance of activity A^, with respect to Service S^* It ^s 
assijohed/to be the average of performance for all the executions of A^ for 
S.£* , Also .associatfed vi th each activity is a performance factor p . 
whiQ)i 4s interpreted as ^the £cali^ jto^ he ,applied. ,to. the (i)' 

to obtain the level of performance for the jth activity which minimizes 
the goal programming objective functipn*. It should be noted that the 
problem of identifying a "good" level of performance for activities,. a*.e* 
deteraihing the appropriate values of the R_, (i) \s is compounded by the 
^multiple use, of the activity by different and possibly conflicting 
services. Therefore, the modification of ah activity's processing rate 
to achieve one goal may be counter productive to the attainment of another 
goal. It is to this possibility of multiple conflicting interactions and 
goals, that this methodology is focused. 

~3T^Formula tlon'^of '"the^S t a'g^e"^2'^Eva lu Itlive "Pr oceaiir e — 

Stage 2 is based on an evaluative procedure commonly called-mul.tiple. 
goal programming (MGP) . The procedure was first formulated by Charnes 
and Copper (CHA61) in 1961 to solve linear programming problems that had 
conflicting constraints. Ijiri (IJI65) developed the details of the pro- 
cedure within the framework of mathematical programming. This technique 
has been used to solve problems in the ar^as of strategic management 
planning such as accounting control (Ijiri), advertising-media planning 
(Charnes and Cooper (CHA68)), and resource allocation (Lee, (LEE72)). 
The employment of goal programming in conjunction with information system 
perfprmance evaluation is a new use of the procedure. * 

There are three reasons for choosing multiple goal programming for 
use in this stage of the methodology. First, this approach can evaluate 
linear and ordinal multiple goal situations, both of which are inherent 
to information systems evaluation. For example, one user class may pay 
twice as much for its service, and, therefore, satisfaction of its goals. 
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onay be worth twice as much as^ others; a linear relation* On the other 

hand, certain users, such .as a crltlical patient monitoring application, , . 

may have incomparabre importance relative tb;:6ther classes; an ordinal 


s 


I* 

f 


relation. Second, multiple goal programming produces a solution that hot 
only evaluates the total goal situation, but also evaluates each goal. 
Individually. One of the purposes of this methodology Is: to determine 
. ±he .critical :user classes .and\associated ^activities . Third, MGE' der 
-rives the- "best" design under- the given goal cblis^^ Other design 
approaches such ♦as weighting, sequential eriminatlon, and spatial prox- 
imity (MAC73), are based oh selecting the "best" design from a finite 
,set of alternatives. The purpose of the overall methodology^ however, 
is. to design ah appropriate^ system to satisfy the user and resource 
constraints. The standard formulation; of a multiple goal progranmiihg 
problem is: 


\ 




:[A] Minimize ' P • D , - - , 
Subject to A • X + D = G 
where 


\ 


? 

\ : 


= 3 matriX"*6f- *technological-coeff-icients-which- c - — 
be thought ;o£ as the. rates at whicK the Ith. 
service^uses the jth resource 
X = the .array of resulting, system resource allocatioh 
levels 

G = the array of service goals 
D = the array of dlscrepahcles from these goals 
^ P » the array of peiialtles associated with the 
d Iscrepancles^ in D 


> 


X 
7 

r _^ ^ 


and where the objective function is to minimize the product of the dlsr 
crepancies and their asfsociated pehaltles. The solution to a multiple 
goal prograramlhg problem represents X set of levels for tha 
resource allocatioh vector X such that the objective funct^oh is mihimlzed. 
the remainder of this section discusses the specific formulation for the 
Stage 2\c6mp6nent of' the methodology. 

Figure; 4 illustrates the relationship between MGP, the information 
system activities, and Its servicing of user requests.,^ The servicing 
of a request type i is. a sequence of activities, A^^i A^, A^, each 
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*Vaiues of Bj and D(i)*s are. the outputs of Goal Program Optimization Problem 
O Figure 4* Relationship Between Goal Programming and Information System Characterization 



assumed to be measurable by I? general, the measure can be a 

function M(Rj(i)) of the service tijne, however, just be 
employed in the ^following discussion. 

The performance of the information system for service type i is 
:giyen by the relation * 

(1) Tg(i) « Z^^^ (R^(i)) 

where Tg (i) is the aver age^ system resp onse ti me to service requests of 
type i. Assuming that the performance goal for the service is T (i), 
then the discrepancy between performance and goal is given by 

(2) D(l) = Tg(i) - Tg<i):. . 

The objective of MGP is to determine new performance levels for each 
activity in such a mahnef that the weighted discrepancyi P • D, is 
minimized (hopefully to zero) . Letting be a scaling factor to be 
applied to the jth activity, then the new performance level for the 
activity is Rj(i) . Incorporating the 's into equation U) results 
in-^the following expression for the discrepancies: 

. i3L ™Jl(i]L^gjCi)^2^jJ(R^ ^^^^ _™ 

Observe that both positive and negative discrepancies are possible, 

and, ther.efore, the formulation of the user goal evaluation as a MGP 
(1) 

problem becomes : 

[B] Minimize P^ • D**" + P^ v d" 

s.t. R + d"" - D**" « G 

where 

R is a matrix of service rates 
P is the array of scaling factors 
G is the array of user goals 
and D are the arrays of, respectively; the positiye 
and negative discrepancies from the iiser goals 
P and P are the arrays of penalties associated with the 

corresponding positive and negative discrepancies. 



The'domplete derivation api>ears in a previous paper by the authors 
presented et the Annual Conference of the Computer Measurement Group, 
.November, 1976 (CHN76) . 
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The. formulation serves two purposes: it evaluates goal achivement and 
produces- the Pj*s. By setting the valxies of the p*s to reflect only the 
current configuration (i.e.," = i) , the evaluation, of the system's 
attainment of the user goalis is accomplished. 

^Experiments with formulation [B] prbdiiced valid, but impractical 
sets of .p*s. The MCP fprobiem as sta tedl. allowed >f or the possibility of 
solutions where a Pj could equal 0^ clearly an unacceptable situation, 
in border to inhibit this type of splution, limits were placed on^the 

i£5Bgjl25i^ipMs^^ Th is was accomplished with the followihg. 

set of .additional constraints: - ^ . — 



where 



0<Lj 5 1 
1 . 



is used to restrict the alternative possibilities, for .the case that 
LjLiJfiliil^fiuJ^ :In generai,_the_ 



LjlJLjfil^ :In general,. 

.setrof /all;positive>discrepancies-for-Lj^sy ^(jXj^r^ix^r^^^jM'J)^* M and 
likewise for H^. Xv^,_V2, . . ..,v^- ^ N^.- (m" and N" have similar definitions). 

These constraints are reflected^ in the objective function in a 
manner different than previous constraints. Instead of minimizing both 
discrepancies, only one is minimized. In tlaef^ase of L, only uT is 
included, since, if |x is driven to zero, then p - |x = L , Implying 
that p > L., the desired condition » Similarily, for only u is in 
the objective function because minimizing results in 
P3.<:H--. 

These added constraints also have a physical interpretation relative 
to the evaluation of the system. No activity can be eliminated from a 
system (i.e^, = 0) since must be greater than 0. In general, how- 
represents the lower bound on the degree of reduction feasible 



ever. 



for the current level of usage for an activity. For example, - .25 
Implies .that the usage time for activity j can be made, at most, fourutimes 
shorter, being reduced to 25% of its current level. Similarily, repre- 
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sents the upper bound on the degree to which an activity* s level can be 
Increased (made longer) • It must be emphasized that these limits are 
onlyi rough estimates, not exact values* 

In order to reduce the number of alternatives one should minimize 
the number of modifications indicated per evaluation iteration. Since 
modifications are character ized^ by the production of Pj's not equal to 
1, a secondary objective of Stage 2 is^ to produce as few 1 solutions 

as possible.. _ This, ^is_acc_oinplished3y^,including.^the_cons train t* . equation^. „ 



while minimizing both and 

Constraints of thiis type provide a default value of 1. for t:he multiple 
goal programming procedures in the case where an activity is neither 
critically inefficient or excessive, (Note: (e^^, e2»*'*»^j^ 

As a result of these added constraints, the actual formulation of 
the MGP problem used in Stage 2 is given in formulation [C] below: 

[C] Minimize * * +.pr..«.-^M~-+^^pi— -N^ 



^ ^^:^e:^ f ^^^^ — 

s.t* R • p + d" - d"** ^ G 

p + - tf** = L 
P + N" - n"** = H 
p + E" - e"^ = 1 ^ 

where 

R. is the matrix of service rates 
p is the array of scaling factors 
G, L, H, and 1 are the- arrays" of goals for the user criteria 
and the respective p constraints 

+ + + 4* 

N- and E- are the arrays of positive and negative dis- 



crepancies from the. respective goals 
are the arrays 
discrepancies* 



+ + 4* *f 

P-, Pjj, and P=; are the arrays of penrlties for the associated 
D M N £» 
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I 

The solution variables for the MG? problem are the p*s. They identi 
fy those activities that must be altered in order to Improve user based 
or system based performance. If the value for a = 1 then the service 
_ , 5^55,^1,^5 of activity j were adequate to satisfy all the user^s 
criteria. If a < i, this implies that the current service rate of 
activity j is insufficient to meet the system's needs. The new service 
irate for the activity should be R,(0 = (pj * (R.(0). If a B > 1 

jyien the current ser.vice xate of. activity j is faster than necessary 

-and^^there exists the-^possibiiity of excess-capacity; The new unit rate- 
should be Rj(-) = (pj) * (Rj(O). 

Assuming that an activity follows the characterization in Figure 3, 
then the analyst has three avenues of action when a 6, 9^ l.\ First, he 
can analyze the queue dispatching discipline in order to increase queue 
throughput (or possibly replace it with a simpler one if p^ > 1) . 
Second, he can alter the service rate characteristics of'^the servers, 
e.g., slower hardware devices. And third, he can increase (decrease) 
the degree of parallelism among servers, for example, by adding (re- 
moving) a second channel, controller, etc. 

4. Liaison With Stage 1 

The critical factors in the Stage 2 evaluation are the values for 
the Rj(i)*s needed by the MGP formulation. These values are calculated 
In Stage 1 and are the statistical measure produced vis-a-vis the simu- 
lation. The specific model to be evaluated is ,.the result of tha 
heuristic procedures constituting Stage 3. The liaison is based upon the 
assumption that an information system can be viewed as a collection of 
resource allocation and task management activities and user oriented 
services. This view is supported by the literature, e.g., Madnick (MAD74) 
and Zurcher and Randall (ZUR69) . IPSS also views the modeling of an 
information system in a similar manner, and thus, facilitates the 
development of the formal liaison with the MGP user evaluation. 

The view of the system taken in Stage 2 (as illustrated in^ Figure 2) 
has an analog in IPSS. Its basic modeling concept is that of a service 
as shown in Figure 5. The service is classified in IPSS as a procedural 
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Figure 5. Functions of IPSS Service Entitles 
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facility and is capable of representing any information system activity 
including request (job) scheduling, task management, resource allocation, 
secondary storage I/O processing and application software. Since services 
,are allocatable facilities in IPSS they have associated with them both 
queueing and utilization statistics. Furthermore, service behavior can 
be predicated on the quantity and characteristics of other IPSS hardware 
and software facilities. Therefore, the statistics associated with service 
facilities have the appropriate structure to service as the Hj*s needed 
in -^Stage 2. 

To complete the formal liaison between the two stages a second 
feature is employed. This is the Task facility. Through its use, the 
service facility statistics can be automatically segregated into service 
statistics by user. In this manner, the statistic (i) , required by 
Stage 2, is collected. Tfiiis, the Stage 2 users (indexed by i) and the 
activities (indexed by j\ are, respectively, an IPSS model's Task and 
Service facilities. An Rj(i) is the sum of queueing and utilization 
statistics gathered for Service i when executing Task j. 

Stage 1 must also be adaptive to model changes dictated via Stage 3. 
Again, the IPSS model synthesis philosophy and language constructs permit 
the^-desired -^daptlveness;; — This -i-s""po*^sl5'ib:rerf or^reaso^ ^foo aetailed toT 
discuss in this paper^ A complete description of IPSS is available in 
the document titled "The Information Processing System Simulator (IPSS): 
Language ^Syntax and Semantics" (DEL76) . Briefly, however, possible 
modifications to an existing and executing model without requiring complete 
reformulation include changes to:* 

1. timing and space characteristics associated with secondary 
storage hardware and storage media, 

2. the secondary storage I/O configuration, 

3. user usage patterns and service requirements, 

4. file organization methods and space management policies, 

5. the queueing disciplines associated with job scheduling, re- 
source allocation and task management, and 

6. memory management policies ♦ 

IPSS supplies the Stage 1 processing with a capability of being 
self-adaptive with respect to Stage 3 outputs ♦ Currently, the methodology 
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mis- 



employs modeler assistance in Stage 3. Future research will be directed 
at :providing morfe sophisticated heuristics: for Stage 3 in. order to 
provide a truly self-contained iterative methodology for the multiple 



criteria, evaluation.,of^information*.systems-« 



5. Stage 3 Analysis 



the current configuration is satisfactory and to formulate a new model 
in light of the data provided from Stage 1 and Stage 2. Figure 6 shows 
the information flow to Stage 3. New models reflect the performance 
goals of bcth the user and the system. Heuristics using Sutherland's 
definition of a heuristic are employed in Stage 3. 

The problems encountered are complex and unstructured. Determining 
if the current design is acceptable requires a mixture*, of objective and 
subjective reasoning. It would be a rare situation if all the user goals 
and system constraints were satisfied simultaneously. Generally, an 
extrmely wide spectrum of acceptable performance levels and alternative 



■^dlgsigp~are"'po^^^ 

criteria. Trade-offs will dominate the decision processe^^^^ J^ny factors 
affecting suitable designs may not be included in the formulations and 
procedures of the first two stages. For example, there may be external 
political:, organizational or economic considerations that are not directly 
related to the performance of the system, but may be a major factor in^the 
final decision. The methodology does assume, however,, that the heuristic 
procedures have access to this external criterion. 

When it has been determined that another iteration is desirable, 
it is assumed that the heuristics will examine current and past designs. 
Whatever the heuristic employed, ideally its objective is to produce a 
sequence of models whose characteristics' (for all p*s) behave as 
follows: 



(1) 



A heuristic is "a disciplined trial-and-error process,..., an exer- 
cise in successive Improvement, where we may learn from both success 
and failure and where the criteria for success and failure may vary 
with wtiat we have previously learned** (p. 183, (SUT75)).. 




Figure 6. Information Flow to Stage 3 
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The emphasis of the current research is to provide insight- into the 
decision .process for improving the performance of information, systems . 
Stage 3 is this decision procesis. It is aided by input from four 
sources .T^thin the methodology. These -sources are; Xa); the Stage d 
outputs, specifically the p^'s identified to improve user arid system 
,,gpjal performance,, (b) system behavior statistics from Stage 1,. (c) the 
current modal, and (d), historical data from prior iterations, . It should 
be emphasized that at this juncture in the development of the methodology 
no formal heuristiq procedures have been implemented. It is one of the 
purposes of this research, however, to investigate the appropriateness 
and success of various heuristic decision rules. Rules of thumb such 
as those proposed by Buzen (BUZ76) are poss'ible avenues to be investi- 
gated. 



6. An Example 



In order to validate the procedures developed in this methodology 
and the liaison between Stage 1 and Stajge 2, ,a test case was developed-; 
This example was modeled arid executed in IPSS to satisfy the Stage 1 
requirements. Several iterations were applied, demonstiratirig the eval- 
uative capabilities of the methodology. The following is a description 
of the problem, the corresponding model and the results of the first 
two iteratioris. 

The ex^ple is a model of an on-line document retrieval system, 
there are three files associated with the system; an author/title 
index (A/t) , a system documerit identif icatiori file (ID) , and the docu- 
ment file itself tDOC) . " They are structurally reiated..such that an entry 
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In the A/T file points to one or more entries in the ID file and each 
ID file entry* in turn^ is associated with only one DOC entry. 

The model is designed so that a unique activity is associated with 
the accessing of each file; Activity 1 with the A/T file, Activity 2 with 
the ID file and Activity 3 with the DOC file. Each activity performs 
similar functions: obtaining- and releasing devices, reading records, 
performing I/O techniques, but to different files. 

It is assumed that the system supports three user classes,, each 
with -a diff erent d and each, characterized^ - 

by a, different combination of activities. The purpose of the first user, 
class, USER^, is to retrieve a document for a particular author, thus 
utilizing all three activities* Those in the second user class, USER2, 
want .to determine the 'existence/non-existence of an entry in the system 
for a given author, and therefore, need, to use. only Activity 1. The 
final user class, USER^, already has the address of the ID entry and 
wants to retrieve the associated DOC entry requiring only Activity 2 
and Activity 3. 

In order to complete the formulation of the performance evaluation 
problem for this methodology, assumptions concerning the performance of 
the system were /made. A summary of these assumptions for the user goals 
and p constraints and the penalties associated with the corresponding 
discrepancies in accordance to the requirements of formulation [C] is 
shown below. 



G 




( 150.0, 


L 




< ■•2. 


H 




( 5.0, 


T 




( 1.0, 






(1000.0, 






( 1.0, 






< 1.0, 



100.0, 400.0) 

.2, .2) 

5.0, 5.0) 

1.0, 1.0) 

10.0, 1000.0) 

0.0, 0.0) 

1.0, 1.0) 



The model constructed in IPSS assumed a simple configuration of 
one processoi:< and one bank of IBM 2314 type direct access devices. 
Under a given loading (which is not a controiiable variable in this . - 
methodology) the resulting performance statistics, the values of matrix 
R, are shown beiow. 



.0.0 



Sj^(l) = 37.2 



37.2 



Q2(l) = 0.0- 



R2(i) 



37.1 
37.1 



Q3(l) 



11.4 



S^Cl) = 90.9 
R^Cl) = 102.3 



Qj^^(2) = .12.2: 
S^ii) = 36.3 



Rj^^(2) = 48 .5 



Q2X3) = 0.0 
$2(3) = 39.1 
R2(3) = 39VT 



^3(3) 
83(3) 



8,9 
93.5 



13(3); = 102.4 



This performance information, coupled with the goal assumptions, 
was input to Stage 2. The evaluation of the current configuration's 
performance with respect to the set of user's goals is shown in Table 1. 
It indicates that the goals of user classes 2 and 3 were satisfied with 
a good margin of 'slack; (which is not penalized in this example) while 
the goal of the first user clasj was not satisfied (under-achievement 
Implying non-satisfaction) . 



Table 1 Uster Goal_,Evalu^^^ _ 



Goal 


Under -Achievement 


Over -Achievement 


USERj^ 


26.6 


0.0 






USER2 


0.0 


•51.5 


USER^ 


0.0 


258.5 



In the second phase of Stage 2, the pi*s are allowed to be manipu- 
lated until they satisfy the user goal constraints and best sacisf ice* 
the system guideline constraints^. The result is -the identification of 
those, activities whose performance can be,: arid needs to be^ improved 
with respect to one oir? both of the criteria. The values for the pis 



as calculated were 



Pi = 1-P ' -P3 = 

These are interpreted as indicating that both Activity 1 and Activity 2 
were adequate to meet the user- demands put to them. Activity 3, however, 
^was found to be insufficient to satisfy the requirements of user 
classes 1 and 3. The; modification indicated is to reduce the present 
level of usage for Activity* 3 by at least 1/4 in. order to satisfy the 
user goals, in particular^ the first usrer class goal. 

The determination of whether to cease the design loop by accepting 
this performance or to continue by mgdify^ing the existing model is made 
in Stage 3.. Given the stated goal/penalty structure, it was assumed 
that.:the. under-achievement of USERi goal was at an unacceptable level and 
the design process must continue if possible. By examining the queueing 
and service time statistics for the first iteration, one can, eliminate 
some of the modification possibilities. The result of Stage 3 analysis 
was a decision to replace the IBM 2314 type device with a faster one, 
i.e;, an IBM 3330 type device. 

The original model of this example system was dynamically altered 
to reflect this modification. Under the same Ibading as before, the 
following performance statistics were accumulated. 
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= 3.8 


s^Cl) 


= 11.9 


82(1) 


= 11.3 


83(1) 


= 27.6 


R^d) 


= 11.9 




= 11.7 


R3(l) 


= 31.4 


Ql(2) 


= 3.8 


Q2(3) 


= 0.0 


Q3(3) 


= 0.0 


8^(2) 


= 11.1 


82(3) 


= 12.0 


83(3) 


= 29.1 


\(2) 


14.9 


R2(3) 


= 12.0 


R3(3) 


= 29.1 



Stage 2 analysis showed that now all three user class goals were 
satisfied (i.e., over-achieved). The calculation of the p's, how- 
ever, indicated that while Activities 1 and 2 were still adequate 
(Pj^ « P2 * Activity 3 how had the possibility of excess capacity 

(Pj, « 4.0). Although a slower and probably less expensive device for 



Activity 3 would be more appropriate,: we had found In the first Iteration 
thai such a device was: not able to satisfy all the user goals. There-- 
fore, In future iterations. Stage 3 procedures had* to eacamlrie more 
subtle methods of imprpving-performahce. 



7* Conclusion 




Modern information systems do not exist as entities unto themselves, 
but must interact with their environment, i.e., their users. The loading 
and mix of the users affect the performance of the system resources and 
likewise, the service characteristics of the system resources affect 
the satisfaction of user goals. In order to design such systems, one 
must satisfy a large set of users demanding a conflicting set of per-- 
formance goals while operating within efficiency and minimum cost con- 
straints; Thus, performance evaluation of information systems is a 
multiple criteria problem. Concurrently, satisfying both of these sets 
of criterlk is the goal of this methodology. Current available techniques* 
however, only address one side of the problem, either the user or the^ 
system. The methodology described in this paper establishes a formal 
liaison between the evaluation of user goals as a function of system 
behavior and the analysis of system resource performance as a function, 
of user demand, '^thereby, facilitating muiti-criteria evaluation. 
.... .jrhe_methodology_is Ate separate but 

integrated stages. The first stage models and evaluates system behavior. 
The particular technique employed in the first stage is IPSS and it 
is able to, .collect the necessary statistic, R^Ci) . The second stage 
evaluates the user based criteria and provides evaluative insight into 
performance improvement. Solution of the MGP formulation in [C] pro- 
duces a set of P's, the variables of Stage 2 which indicate inefficiencies 
and/or excesses in the current model. And finally', the third^stage 
heufistically determines the current model's acceptability and need 
for modifications. 

The evaluative procedures developed for this methodology have been 
shown to be valid in practice. Furthermore, this methodology provides 
an excellent basis for continued-research. into areas such as: 
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investigation iiito the causal relationships between user i 
demand and system/activity, 
b- sensitivity analysis of these relationships, y 

c. investigation into suitable heuristics for Stage 3, either 
testing existing heuristics or development of new ones, and 

d, development o£ . heuristic /modification rules to close the 
design loop into an automatic self-modifying process. 

f ' ' 
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